Skip to content

results(summary): rename cases to tests, trim run metadata - #1697

Merged
christso merged 2 commits into
mainfrom
cpl5-summary
Jul 6, 2026
Merged

results(summary): rename cases to tests, trim run metadata#1697
christso merged 2 commits into
mainfrom
cpl5-summary

Conversation

@christso

@christso christso commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Renames run-root summary.json's cases[] to tests[] and aligns counts to test/sample terminology: total_cases/passed_cases/failed_casestotal_tests/passed_tests/failed_tests, total_instances/errored_instancestotal_samples/errored_samples.
  • Drops the metadata.run_id duplicate of the summary root's run_id field. summaryRunId() (used when rebuilding cross-run indexes) reads the root field first and falls back to the old metadata.run_id location so historical bundles written before this change still resolve correctly.
  • Renamed internal variables in buildRunSummaryArtifact (casesByKeytestsByKey, instancessamples, etc.) to match the new terminology.

Implements av-cpl5.2 (child of source-of-truth tracker av-cpl5). This is the writer side only — RunSummaryArtifact, buildRunSummaryArtifact, writeInitialRunSummaryArtifact, and aggregateRunDir's summary-writing path in packages/core/src/evaluation/run-artifacts.ts. Dashboard/CLI reader compatibility for historical bundles (old cases/total_cases names) is scoped to the parallel bead av-cpl5.1; no reader code outside run-artifacts.ts referenced these fields (verified via repo-wide search), so nothing else needed updating in this PR. .internal/index.jsonl and IndexArtifactEntry (av-cpl5.3's territory) were left untouched.

Known follow-up (out of scope here): metadata.eval_file is written as whatever absolute/relative path the CLI passes into buildRunSummaryArtifact/writeArtifactsFromResults — in a live dogfood run it came through as an absolute host path, while the sibling metadata.runtime_source.eval_files is already repo-relative (normalized upstream in run-eval.ts). Fixing eval_file would need cwd/repoRoot plumbing through multiple caller files outside this bead's declared scope; flagging for a follow-up bead rather than expanding this PR's surface.

Test plan

  • bun test apps/cli/test/commands/eval/artifact-writer.test.ts — 75 pass, including new/updated assertions for tests[], counts.total_tests/passed_tests/failed_tests/total_samples/errored_samples, and top-level run_id.
  • Regression: bun test apps/cli/test/commands/eval/aggregate.test.ts packages/core/test/evaluation/target-execution-artifacts.test.ts packages/core/test/evaluation/orchestrator.test.ts — 115 pass.
  • Regression: bun test apps/cli/test/commands/results/report.test.ts apps/cli/test/commands/results/export.test.ts — 39 pass.
  • bun --filter @agentv/core typecheck — clean.
  • bunx biome check on touched files — clean.
  • Live dogfood: bun apps/cli/src/cli.ts eval run examples/features/rubric/evals/operators.eval.yaml --target llm --workers 1 --threshold 0.8 against a live Azure LLM target with a live llm-rubric grader (threshold 0.8, not a 0-threshold smoke check) — passed 100%, and the resulting .agentv/results/<run_id>/summary.json was inspected end-to-end to confirm the new tests[]/counts.total_tests/etc. shape and the removed metadata.run_id duplicate.

🤖 Generated with Claude Code

Run-root summary.json now uses tests/test terminology instead of
cases/case: counts.total_cases/passed_cases/failed_cases become
total_tests/passed_tests/failed_tests, total_instances/errored_instances
become total_samples/errored_samples (sample-level counts alongside the
new test-level ones), and the cases[] array is now tests[]. Also drops
the metadata.run_id duplicate of the summary root's run_id field, with
summaryRunId() reading the root field first and falling back to the old
metadata location for historical bundles.

av-cpl5.2
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Jul 6, 2026

Copy link
Copy Markdown

Deploying agentv with  Cloudflare Pages  Cloudflare Pages

Latest commit: e1f2aa4
Status: ✅  Deploy successful!
Preview URL: https://1fb62a04.agentv.pages.dev
Branch Preview URL: https://cpl5-summary.agentv.pages.dev

View logs

The programmatic API test asserted summary.metadata.run_id, which moved
to the summary root in the cases->tests rename. Update the test's inline
type and assertion to match.

av-cpl5.2
@christso
christso merged commit a983924 into main Jul 6, 2026
8 checks passed
@christso
christso deleted the cpl5-summary branch July 6, 2026 09:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant